Systems, devices, and methods for distributed artificial neural network computation

ABSTRACT

Robots and robotic systems and methods can employ artificial neural networks (ANNs) to significantly improve performance. The ANNs can operate alternatingly in forward and backward directions in interleaved fashion. The ANNs can employ visible units and hidden units. Various objective functions can be optimized. Robots and robotic systems and methods can execute applications including a plurality of agents in a distributed system, for instance with a number of hosts executing respective agents, at least some of the agents in communications with one another. The hosts can execute agents in response to occurrence of defined events or trigger expressions, and can operate with a maximum latency guarantee and/or data quality guarantee.

BACKGROUND Technical Field

This disclosure generally relates to the field of artificial neuralnetworks including artificial neural networks used in distributedcomputing and/or associated with operation of robotics.

Description of the Related Art

Machine Learning

A computer, which is a machine, can perform or succeed at one morerelated tasks as defined by a measure. The computer learns if afterexposure to information characterizing an event the computer improvesunder the measure at performing the one or more related tasks. Further,the computer learns without updates to any processor-executableinstructions by imperative programming.

Artificial Neural Networks

Artificial neural networks (ANNs) are a family of machine learningmodels inspired by biological neural networks. The models includemethods for learning and associated networks. A network includes unitsconnected by links and is similar to a graph with units in place ofnodes and weighted edges in place of unweighted edges. A unit has anon-linear response to an aggregate value for a set of inputs. Theinputs are the responses of other units as mediated by weighted edgesand/or a reference or bias unit. Traditionally and to an immaterialscaling factor a unit has as its state, a value between and includingminus one and one, or zero and one. The term unit distinguishesartificial neural networks from biological neural networks where neuronsis a term of art. These ANN are simulated using techniques fromtechnical fields including computing, applied statistics, and signalprocessing. The simulation causes an ANN to learn over input, or applythe learning. That is, the simulation effects an evolution or adaptationof weights for the units and edges. These weights, also calledparameters, are used in later computing including when simulating theANNs' response to inputs. Both learning and application of learning ofANN require computational resources of at least time andprocessor-readable storage media space. There are many technicalproblems which prevent various systems, devices, articles, and methodsfrom operating ANN with reduced resources.

BRIEF SUMMARY

A computing system including back propagation substantially as describedand illustrated herein.

A system including a computer and a robot substantially as described andillustrated herein.

A robotic apparatus substantially as described and illustrated herein.

A method of operation of a computing system substantially as describedand illustrated herein.

A method of operation of an artificial neural network, including aplurality of variables and a plurality of parameters, may be summarizedas including computing, by at least one hardware processor, a firstplurality of values for the plurality of variables in the artificialneural network based on a first plurality of values for the plurality ofparameters over a first period; calculating, by the at least onehardware processor, a first plurality of incremental errors for theartificial neural network over the first period; generating, by the atleast one hardware processor, a second plurality of values for theplurality of parameters; storing, by the at least one hardwareprocessor, the second plurality of values for the plurality ofparameters; interleaved with calculating the first plurality ofincremental errors for the artificial neural network over the firstperiod, computing, over a second period, by the at least one hardwareprocessor, a second plurality of values for the plurality of variablesfor the artificial neural network based on the first plurality of valuesfor the plurality of parameters; and returning, by the at least onehardware processor, the second plurality of values for the plurality ofparameters.

The method may further include computing, by the at least one hardwareprocessor, over a third period, a third plurality of values for theplurality of variables in the artificial neural network based on thesecond plurality of values for the plurality of parameters; calculating,by the at least one hardware processor, a second plurality ofincremental errors for the artificial neural network over the thirdperiod; generating, by the at least one hardware processor, a thirdplurality of values for the plurality of parameters; and interleavedwith calculating the second plurality of incremental errors for theartificial neural network over the third period, computing, by the atleast one hardware processor, over a fourth period, a fourth pluralityof values for the plurality of variables for the artificial neuralnetwork based on the second plurality of values for the plurality ofparameters.

The method may further include storing, by the at least one hardwareprocessor, the first plurality of values for the plurality of variablesfor the artificial neural network.

The artificial neural network may be associated with an objectivefunction, the method further including computing, by the at least onehardware processor, a gradient for the objective function with respectto the plurality of parameters; and performing, by the at least onehardware processor, gradient descent with the gradient to create thesecond plurality of values for the plurality of parameters. Computingthe plurality of gradients for the objective function with respect tothe parameters may include unfolding, by the at least one hardwareprocessor, the artificial neural network over a duration less than orequal to the first period; and performing, by the at least one hardwareprocessor, backpropagation through time on the unfolded artificialneural network.

The method may further include receiving, by the at least one hardwareprocessor, a plurality of input values for the artificial neuralnetwork.

The method may further include producing, by the at least one hardwareprocessor, a plurality of output values for the artificial neuralnetwork; and causing, by the at least one hardware processor, a robot tomove based on the plurality of output values.

The method may further include computing, by the at least one hardwareprocessor for another artificial neural network that includes anotherplurality of variables and another plurality of parameters, a firstplurality of values for the other plurality of variables of the otherartificial neural network based on a first plurality of values for theother plurality of parameters; running, by the at least one hardwareprocessor, the other artificial neural network backward over the firstperiod; generating, by the at least one hardware processor, a secondplurality of values for the other plurality of parameters; andinterleaved with computing, for the other artificial neural the firstplurality of values for the other plurality of variables of the otherartificial neural network, computing, by the at least one hardwareprocessor, a second plurality of values for the other plurality ofvariables for the other artificial neural network over the secondperiod.

Behaviors of the artificial neural network may be described by a systemof differential equations, the method further including numericallyintegrating, by the at least one hardware processor, the system ofdifferential equations. Generating the second plurality of values forthe plurality of parameters may include identifying, by the at least onehardware processor, a system model for a robot; and returning, by the atleast one hardware processor, a system model for the robot.

A system may be summarized as including at least one processor; at leastone non-transitory computer-readable storage media communicativelycoupled to the first processor, and which stores at least one ofprocessor-executable instructions or data thereon which when executedcauses the at least one processor to: define an artificial neuralnetwork including a plurality of parameters and a plurality ofvariables; compute, over a first period, a first plurality of values forthe plurality of variables based on a first plurality of values for theplurality of parameters; calculate a first plurality of incrementalerrors for the artificial neural network over the first period; generatea second plurality of values for the plurality of parameters; andinterleaved with calculation of the first plurality of incrementalerrors for the artificial neural network over the first period, compute,over a second period, a second plurality of values for the plurality ofvariables based on the second plurality of value for the plurality ofparameters.

When executed, the processor-executable instructions may further causethe at least one processor to compute, over a third period, a thirdplurality of values for the plurality of variables based on the secondplurality of values for the plurality of parameters; calculate a secondplurality of incremental errors for the artificial neural network overthe third period; generate a third plurality of values for the pluralityof parameters; and interleaved with calculation of the second pluralityof incremental errors for the artificial neural network over the thirdperiod, compute, over a fourth period, a fourth plurality of values forthe plurality of variables based on the fourth plurality of values forthe plurality of parameters.

The artificial neural network may be associated with an objectivefunction, and when executed, the processor-executable instructions mayfurther cause the at least one processor to: compute a gradient for theobjective function with respect to the plurality of parameters; andperform gradient descent with the gradient creating the second pluralityof values for the plurality of parameters.

When executed, the processor-executable instructions may further causethe at least one processor to: unfold the artificial neural network overa duration less than or equal to the first period creating an unfoldedartificial neural network; and perform backpropagation through time onthe unfolded artificial neural network.

The system may further include a sensor subsystem coupled to the atleast one processor; and where, when executed, the processor-executableinstructions further cause the at least one processor to: receive aplurality of input values for the artificial neural network from thesensor subsystem.

The system may further include a robot coupled to the at least oneprocessor; and where, when executed, the processor-executableinstructions further cause the at least one processor to: produce aplurality of output values for the artificial neural network; and causethe robot to move based on the plurality of output values.

The system may further include a network interface subsystem coupled tothe at least one processor; and where, when executed, theprocessor-executable instructions further cause the at least oneprocessor to: send the plurality of variables of the artificial neuralnetwork; and send the plurality of parameters of the artificial neuralnetwork.

The system may further include a network interface subsystem coupled tothe at least one processor; and where, when executed, theprocessor-executable instructions further cause the at least oneprocessor to: receive the plurality of variables of the artificialneural network; and receive the plurality of parameters of theartificial neural network.

A system may be summarized as including at least one processor; at leastone non-transitory computer-readable storage media communicativelycoupled to the first processor, and which stores at least one ofprocessor-executable instructions or data thereon which when executedcauses the at least one processor to: define an artificial neuralnetwork including a first part and a second part, the first part of theartificial neural network comprising a first plurality of weights, thesecond part of the artificial neural network comprising a secondplurality of weights, and a plurality of variables; operate the firstpart of the artificial neural network in a forward direction for a firstperiod according to a first plurality of values for the first pluralityof weights; operate the first part of the artificial neural network in abackward direction over the first period according to the firstplurality of values for the first plurality of weights; update the firstplurality of weights for the first part the artificial neural network toa second plurality of values for the first plurality of weights;interleaved with operation of the first part of the artificial neuralnetwork in the backward direction over the first period, operate thefirst part of the artificial neural network in the forward direction fora second period according to the second plurality of values for thefirst plurality of weights; operate the second part of the artificialneural network in the forward direction for the first period accordingto a first plurality of values for the second plurality of weights;operate the second part of the artificial neural network in the backwarddirection over the first period according to the first plurality ofvalues for the second plurality of weights; update the second pluralityof weights for the second part the artificial neural network to a secondplurality of values for the second plurality of weights; and interleavedwith operation of the second part of the artificial neural network inthe backward direction over the first period, operate the second part ofthe artificial neural network in the forward direction for the secondperiod according to the second plurality of values for the secondplurality of weights. The at least one processor may include a firstprocessor; and the processor-executable instructions when executed maycause the first processor to: operate the first part of the artificialneural network in the forward direction for the first period accordingto the first plurality of values for the first plurality of weights; andoperate the first part of the artificial neural network in the forwarddirection for the second period according to the second plurality ofvalues for the first plurality of weights. The at least one processormay include a second processor; and the processor-executableinstructions when executed may cause the second processor to: operatethe first part of the artificial neural network in the backwarddirection over the first period according to the first plurality ofvalues for the first plurality of weights; and update the firstplurality of weights for the first part the artificial neural network tothe second plurality of values for the first plurality of weights. Theat least one processor may include a third processor; and theprocessor-executable instructions when executed may cause the thirdprocessor to: operate the second part of the artificial neural networkin the forward direction for the first period according to the firstplurality of values for the second plurality of weights; and operate thesecond part of the artificial neural network in the forward directionfor the second period according to the second plurality of values forthe second plurality of weights. The at least one processor may includea fourth processor; and the processor-executable instructions whenexecuted may cause the second processor to: operate the second part ofthe artificial neural network in the backward direction over the firstperiod according to the first plurality of values for the secondplurality of weights; and update the second plurality of weights for thesecond part the artificial neural network to the second plurality ofvalues for the second plurality of weights.

The first artificial neural network may be associated with an objectivefunction, and when executed, the processor-executable instructions mayfurther cause the at least one processor to: perform backpropagationthrough time to the first artificial neural network creating a gradientfor the objective function with respect to the plurality of weights; andperform gradient descent with the gradient for the objective functioncreating the second plurality of values for the plurality of parameters.

The system may further include a sensor subsystem coupled to the atleast one processor; and where, when executed, the processor-executableinstructions further cause the at least one processor to: receive aplurality of input values for the artificial neural network from thesensor subsystem.

The system may further include a robot coupled to the at least oneprocessor; a motion subsystem included in the robot; and where, whenexecuted, the processor-executable instructions further cause the atleast one processor to: produce a plurality of output values for theartificial neural network; and cause, via the motion subsystem, therobot to move based on the plurality of output values.

The system may further include a network interface subsystem coupled tothe at least one processor; and where, when executed, theprocessor-executable instructions further cause the at least oneprocessor to: send the plurality of variables of the artificial neuralnetwork; and send the first plurality of weights and the secondplurality of weights.

The system may further include a network interface subsystem coupled tothe at least one processor; and where, when executed, theprocessor-executable instructions further cause the at least oneprocessor to: receive the plurality of variables of the artificialneural network; and receive the first plurality of weights and thesecond plurality of weights.

The system may further include a user interface, coupled to the at leastone processor; and where, when executed, the processor-executableinstructions further cause the at least one processor to: receive, fromthe user interface, input for the first part of the artificial neuralnetwork.

The system may further include a user interface, coupled to the at leastone processor; and where, when executed, the processor-executableinstructions further cause the at least one processor to: receive, fromthe user interface, input to parameterize an objective function fortraining the first part of the artificial neural network.

A method of operation of a distributed system, including a first hostand a second host, may be summarized as including causing, via at leastone hardware processor, the execution an application including a firstagent, a second agent, and a plurality of parameters, wherein the firsthost executes the first agent, and the second host executes the secondagent; causing, via the at least one hardware processor, the first agentto send a message to the second agent; updating, via the at least onehardware processor, a history of agents data structure with informationthat defines a first set of temporally ordered local connections for thefirst agent and information that defines a second set of temporallyordered local connections to the second agent; forming, via the at leastone hardware processor, a part of an inverse application to theapplication from at least the history of agents data structure, thefirst agent, and the second agent; causing, via the at least onehardware processor, the evaluation of the part the inverse application;and updating, via the at least one hardware processor, the plurality ofparameters of the application.

The method may further include receiving, via the at least one hardwareprocessor, a system specification document defining a plurality ofagents and the plurality of parameters, the plurality of agentsincluding the first agent and second agent.

The method may further include updating, via the at least one hardwareprocessor, a directed acyclic graph included in the history of agentsdata structure. Causing the execution of an application may includeoperating, via the at least one hardware processor, an artificial neuralnetwork in a forward direction.

The method wherein the artificial neural network may be a recurrentneural network and the method may further include numericallyintegrating, via the at least one hardware processor, a system ofdifferential equations that define a plurality of activations for therecurrent neural network. Causing the execution of the part the inverseapplication may include operating, via the at least one hardwareprocessor, an artificial neural network in a backward direction.Operating an artificial neural network in a backward direction mayinclude performing, via the at least one hardware processor, backpropagation through time on the artificial neural network.

The method may further include producing, via the at least one hardwareprocessor, a plurality of output values for the application; andcausing, via the at least one hardware processor, a robot to move basedon the plurality of output values.

The method may further include receiving, via the at least one hardwareprocessor, information that characterizes an event; processing, via theat least one hardware processor, the information that characterizes theevent; and directing, via the at least one hardware processor, the firsthost to execute the first agent based on the presence of the informationthat characterizes the event. Causing the first agent to send themessage to the second agent may include receiving, via the at least onehardware processor, information that characterizes an event; processing,via the at least one hardware processor, the information thatcharacterizes the event; and directing, via the at least one hardwareprocessor, the first agent to send the message to the second agent basedon the presence of the information that characterizes the event.

The method may further include establishing, via the at least onehardware processor, the distributed system with a maximum latencyguarantee.

A system may be summarized as including at least one processor; at leastone non-transitory computer-readable storage media communicativelycoupled to the first processor, and which stores at least one ofprocessor-executable instructions or data thereon which when executedcauses the at least one processor to: execute an application including afirst agent, a second agent, and a plurality of parameters; cause thefirst agent to send a message to the second agent; update a history ofagents data structure with information that defines a first set oftemporally ordered local connections for the first agent and informationthat defines a second set of temporally ordered local connections to thesecond agent; form a part of an inverse application to the applicationfrom at least the history of agents data structure, the first agent, andthe second agent; evaluate the part the inverse application; and updatethe plurality of parameters of the application.

When executed, the processor-executable instructions may further causethe at least one processor to: receive a system specification documentdefining a plurality of agents and the plurality of parameters, theplurality of agents including the first agent and second agent. Thehistory of agents data structure may include a directed acyclic graph.

When executed, the processor-executable instructions may further causethe at least one processor to: operate an artificial neural network in aforward direction.

When executed, the processor-executable instructions may further causethe at least one processor to: numerically integrate a system ofdifferential equations that define a plurality of activations for therecurrent neural network.

When executed, the processor-executable instructions may further causethe at least one processor to: operate an artificial neural network in abackward direction.

When executed, the processor-executable instructions may further causethe at least one processor to: perform back propagation through time onthe artificial neural network.

The system may further include a robot communicatively coupled to the atleast one processor; and wherein, when executed, theprocessor-executable instructions further cause the at least oneprocessor to: produce a plurality of output values for the application;and cause the robot to move based on the plurality of output values.

When executed, the processor-executable instructions may further causethe at least one processor to: receive information characterizing anevent; process the information characterizing an event; and direct thefirst host to execute the first based on the presence of the informationcharacterizing the event.

When executed, the processor-executable instructions may further causethe at least one processor to: receive information characterizing anevent; process the information characterizing the event; and direct thefirst agent to send the message to the second agent based on thepresence of the information characterizing the event.

When executed, the processor-executable instructions may further causethe at least one processor to: establish the distributed system with amaximum latency guarantee.

The system may further include a first host computer, communicativelycoupled to the at least one processor, wherein the first host computerexecutes the first agent; a second host computer, communicativelycoupled to the at least one processor, wherein the second host computerexecutes the second agent; and a communication channel thatcommunicatively couples first host computer and the second hostcomputer, wherein the communication channel has a maximum latencyguarantee.

A method of operation for an artificial neural network, wherein theartificial neural network may include a plurality of parameters, aplurality of input units, a plurality of hidden units, and a pluralityof output units, may be summarized as including providing, via at leastone hardware processor, input to the artificial neural network over afirst time series; computing, by the at least one hardware processor, aplurality of values for the hidden units for the artificial neuralnetwork; computing, by the at least one hardware processor, a pluralityof values for the output units for the artificial neural network over asecond time series; causing, by the at least one hardware processor, therobot to take action based on the plurality of values for the outputunits over the second time series; receiving, by the at least onehardware processor, at least two true output values; evaluating, by theat least one hardware processor, an objective function, wherein theobjective function includes as operands: two or more values from theplurality of values for the output units, and the at least two trueoutput values; and optimizing, by the at least one hardware processor,the objective function with respect to the parameters.

The method may further include returning, by the at least one hardwareprocessor, information characterizing a system identification.

The method may further include defining, by the at least one hardwareprocessor, the first time series in terms of a first frequency.

The method may further include defining, by the at least one hardwareprocessor, the second time series in terms of a second frequency.

The method may further include receiving, by the at least one hardwareprocessor, the objective function.

The method may further include receiving, by the at least one hardwareprocessor, the input from an operator interface.

The method may further include receiving, by the at least one hardwareprocessor, the input from a senor subsystem wherein the sensor subsystemprovides information that characterizes an environment.

The method may further include executing, by the at least one hardwareprocessor, a write agent including a trigger expression, and a bodyexpression; and receiving, by the at least one hardware processor, theinput from the write agent. Reading at least two true output values mayinclude executing, by the at least one hardware processor, a read agentincluding a trigger expression, and a body expression; executing, by theat least one hardware processor, the trigger expression within the readagent; and executing, by the at least one hardware processor, the bodyexpression within the read agent. Computing the plurality of values forthe hidden units for the artificial neural network, and computing theplurality of values for the output units for the artificial neuralnetwork over the second time series, may include operating by the atleast one hardware processor, the artificial neural network in a forwarddirection. Optimizing the objective function with respect to theparameters may include operating by the at least one hardware processor,the artificial neural network in a backward direction.

A method of operation for an artificial neural network, wherein theartificial neural network may include a plurality of input units, aplurality of hidden units, and a plurality of output units, and may besummarized as including providing, via at least one hardware processor,input to the artificial neural network over a first time series;computing, by the at least one hardware processor, a plurality of valuesfor the hidden units over the first time series; producing, by the atleast one hardware processor, a plurality of values for the output unitsover a second time series; receiving, by the at least one hardwareprocessor, at a target output value; evaluating, by the at least onehardware processor, an objective function over the second time series,wherein the objective function includes as operands: the plurality ofvalues for the output units, and the target output value; and optimizingby the at least one hardware processor, the objective function withrespect to the input.

The method may further include causing, by the at least one hardwareprocessor, a body on a robot to move based on one or more of theplurality of values for the output units.

The method may further include returning, by the at least one hardwareprocessor, information characterizing a predictive model.

The method may further include defining, by the at least one hardwareprocessor, the first time series in terms of a first frequency.

The method may further include defining, by the at least one hardwareprocessor, the second time series in terms of a second frequency.

The method may further include receiving, by the at least one hardwareprocessor, the objective function.

The method may further include receiving, by the at least one hardwareprocessor, the input from an operator interface.

The method may further include receiving, by the at least one hardwareprocessor, the input from a senor subsystem, wherein the sensorsubsystem provides information that characterizes an environment.

The method may further include receiving, by the at least one hardwareprocessor, the input from a write agent, wherein the write agentincludes a trigger expression and a body expression; executing, by theat least one hardware processor, the trigger expression within the writeagent; and executing, by the at least one hardware processor, the bodyexpression within the write agent. Producing the plurality of values forthe output units over the second time series may include executing, bythe at least one hardware processor, a trigger expression within a readagent; and executing, by the at least one hardware processor, a bodyexpression within the read agent. Computing the plurality of values forthe hidden units over the first time series, and producing the pluralityof values for the output units over the second time series may includeoperating, by the at least one hardware processor, the artificial neuralnetwork in a forward direction.

A system including a computer and a robot substantially may be asdescribed and illustrated herein.

A method of operation of a system including a computer and a robotsubstantially may be as described and illustrated herein.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

In the drawings, identical reference numbers identify similar elementsor acts. The sizes and relative positions of elements in the drawingsare not necessarily drawn to scale. For example, the shapes of variouselements and angles are not necessarily drawn to scale, and some ofthese elements may be arbitrarily enlarged and positioned to improvedrawing legibility. Further, the particular shapes of the elements asdrawn, are not necessarily intended to convey any information regardingthe actual shape of the particular elements, and may have been solelyselected for ease of recognition in the drawings.

FIG. 1 is a schematic diagram illustrating a portion of a systemincluding distributed hosts that may be used to implement the presentsystems, devices, articles, and methods.

FIG. 2 is a schematic view illustrating an exemplary robot suitable forinclusion in the system of FIG. 1.

FIG. 3 is an elevation view illustrating a robot in accordance with thepresent systems, devices, articles, and methods.

FIG. 4 is a schematic view illustrating an exemplary computer systemsuitable for inclusion in the system of FIG. 1.

FIG. 5 is a flow-diagram of an implementation of a method of operationof distributed artificial neural networks in accordance with the presentsystems, devices, articles, and methods.

FIG. 6A is a schematic diagram illustrating a portion of a system ofagents in accordance with the present systems, devices, articles, andmethods.

FIG. 6B is a schematic diagram illustrating the portion of the system ofagents form FIG. 6A distributed across a plurality of agents inaccordance with the present systems, devices, articles, and methods.

FIG. 7 is a flow-diagram of an implementation of a sequential method foroperation of an artificial neural network in accordance with the presentsystems, devices, articles, and methods.

FIG. 8 is a flow-diagram of an implementation of an interleaved methodof operation of an artificial neural network in accordance with thepresent systems, devices, articles, and methods.

FIG. 9 is a flow-diagram of a method of operation of an agent in adistributed system in accordance with the present systems, devices,articles, and methods.

FIGS. 10-11 are a flow-diagrams of method of operation of a plurality ofdistributed agent in accordance with the present systems, devices,articles, and methods.

FIGS. 12-13 are a flow-diagrams of methods of operation for an agent inaccordance with the present systems, devices, articles, and methods.

FIG. 14 is a flow-diagram of method of operation of an applicationincluding two or more agents in accordance with the present systems,devices, articles, and methods.

FIG. 15 is a flow-diagram of method of operation for systemidentification in accordance with the present systems, devices,articles, and methods.

FIG. 16A is a flow-diagram of method of operation for creatingpredictive models of a system in accordance with the present systems,devices, articles, and methods.

FIG. 16B is a flow-diagram of a worked example from FIG. 16A.

DETAILED DESCRIPTION

In the following description, some specific details are included toprovide a thorough understanding of various disclosed embodiments. Oneskilled in the relevant art, however, will recognize that embodimentsmay be practiced without one or more of these specific details, or withother methods, components, materials, etc. In some instances, well-knownstructures associated with machine learning and/or robotics, such asprocessors, sensors, storage devices, and network interfaces, have notbeen shown or described in detail to avoid unnecessarily obscuringdescriptions of the disclosed embodiments.

Unless the context requires otherwise, throughout the specification andclaims which follow, the word “comprise” and variations thereof, suchas, “comprises” and “comprising” are to be construed in an open,inclusive sense, that is as “including, but not limited to.”

Reference throughout this specification to “one”, “an”, or “another”applied to “embodiment,” “example,” or “implementation” means that aparticular referent feature, structure, or characteristic described inconnection with the embodiment, example, or implementation is includedin at least one embodiment, example, or implementation. Thus, theappearances of the phrases “in one embodiment,” or “in an embodiment,”or “another embodiment” or the like in various places throughout thisspecification are not necessarily all referring to the same embodiment.Furthermore, the particular features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments, examples,or implementations.

It should be noted that, as used in this specification and the appendedclaims, the singular forms “a,” “an,” and “the” include plural referentsunless the content clearly dictates otherwise. Thus, for example,reference to a problem-solving system including “an autonomous device”includes a single autonomous device, or two or more autonomous devices.It should also be noted that the term “or” is generally employed in itssense including “and/or” unless the content clearly dictates otherwise.

The headings provided herein are for convenience only and do notinterpret the scope or meaning of the embodiments.

FIG. 1 shows an exemplary system 100 in accordance with the presentsystem, devices, articles, and method. Various components of system 100are optional. As shown, the system 100 includes a plurality of hosts102-1, 102-2, 102-3, 106-1, 106-2 with two or more of the hosts incommunication with each other. The plurality hosts includes a pluralityof robots 102-1, 102-2, 102-3 (three shown, singularly or collectively102). The robots 102 may be associated with one or more optionaloperator interfaces, such as, operator interface 104. The plurality ofhosts include a plurality of computers 106-1, 106-2 (two shown,collectively 106). While illustrated as three robots 102-1, 102-2, 102-3and two computers 106-1, 106-2 various implementations can include agreater or lesser number of robots 102 and/or computers 106.

The plurality hosts may be organized into host groups, where each hostgroup includes two or more hosts selected from robots 102 and/orcomputers 106. The plurality hosts, e.g., robots 102, and computers 106are communicatively coupled via one or more network or non-networkcommunication channel 108. Examples of a suitable network or non-workcommunication channel 108 include a wire based network or non-networkcommunications channel, optical based network or non-networkcommunications channel, wireless network or non-network communicationschannel, or a combination of wired, optical and/or wireless network ornon-network communications channels. In some implementations, the robots102, operate without a human operator interface 104 or operator, e.g.,autonomously. In some implementations, a human operator at operatorinterface 104 pilots one or more of robots 102. Suitable communicationprotocols include FTP, HTTP, Web Services, SOAP with XML, and the like.The robots 102 operates in, and receives data about, an environment 140that comprises the physical space. The robots 102 can share anenvironment, e.g., robot 102-1 and 102-2, and environment 140-1, orinhabit their own environment, e.g., robot 102-2 and environment 140-2.“About” is employed here in the sense of represent, characterize, orsummarize.

A robot, like robots 102, is an electro-mechanical machine controlled bycircuitry and/or a processor following processor-executableinstructions; a human operator controllable electro-mechanical machine;a robotic subsystem (or apparatus) of another machine including a robot;or the like. A robot performs physical tasks, for example, working withtangible results and/or computational tasks. A robot has the ability tomove in a physical space, such as environment 140, to accomplishphysical tasks. As well, a robot includes computational resources,on-board and/or remote computational resources, to perform computationaltasks. The computational tasks can be in aid of the physical tasks,e.g., planning, as a task, for accomplishing a tangible result tophysical task. A robot has the ability to acquire information fromsensors, on-board and/or remote sensors. A robot can be part of orincluded in a larger system like system 100.

A robot typically includes a propulsion or motion subsystem 152comprising of one or more motors, solenoids or other actuators, andassociated hardware (e.g., drivetrain, wheel(s), treads), to propel therobot in a physical space. The space does not need to be horizontal orterrestrial. Examples of spaces include water, air, vertical spaces,outer space and the like.

A robot typically includes a manipulation subsystem 154 comprising oneor more appendages, such as, one or more arms and/or one or moreassociated end-effectors. An end-effector is a device attached to arobotic arm designed to interact with the environment. End-effectors forrobot operating in unstructured environments are devices of complexdesign. Ideally, these are capable of performing many tasks, includingfor example grasping or gripping or otherwise physically releasablyengaging or otherwise interacting with an object.

A robot includes sensor subsystem comprising one or more sensors, suchas, one or more cameras 156 and/or one or more microphones 158. A sensorsubsystem acquires data for the robots. The data includes environmentalsensor information representative of environmental conditions externalto the robot. As previously noted sensors can include on board sensorsphysically coupled to move with the robot and/or remote sensors whichare separate and distinct from a given robot. In some instances, asensor can be on board a first robot 102-1, and remote from a secondrobot 102-2, while providing information to at least the second robot102-2.

FIG. 2 schematically shows parts of a robot, including a processor, foruse as a host in the system 100, shown in FIG. 1, in accordance with thepresent systems, devices, articles, and methods. In someimplementations, the host is robotic apparatus such as robot 200. Robot200 includes at least one body or housing 202, and a control subsystem203 that includes at least one processor 204, at least one nontransitorycomputer- or processor-readable storage device 208, and at least one bus206 to which, or by which, the at least one processor 204 and thestorage device(s) 208 are communicatively coupled. In someimplementation, a host comprises a sub-set of the illustrated system200, including control system 203, bus 206, storage device 208, andnetwork interface subsystem 210.

Robot 200 includes a network interface subsystem 210 that iscommunicatively coupled to bus 206 and provides bi-directionalcommunicative coupling to other systems (e.g., external systems externalto the robot 200) via a network or non-network communications channel108. The network interface subsystem 210 includes one or more buffers.Network interface subsystem 210 receives and sends data related totraining machine learning models. Network interface subsystem 210 may beany circuitry effecting bidirectional communication ofprocessor-readable data, and processor-executable instructions, forinstance radios (e.g., radio or microwave frequency transmitters,receivers, transceivers), communications ports and/or associatedcontrollers. Suitable communication protocols include FTP, HTTP, WebServices, SOAP with XML, WI-FI compliant, BLUETOOTH compliant, cellular(e.g., GSM, CDMA), and the like.

The robot 200 includes an input subsystem 212 comprising one or moresensors that detect, sensor, or measure conditions or states of therobot and/or conditions in the environment in which the robot operates,and produce or provide corresponding sensor data or information. Suchsensors include cameras or other imagers, touch sensors, load cells,pressure sensors, microphones, meteorological sensors, chemical sensorsor detectors, or the like. The robot 200 includes an output subsystem214 comprising output devices, such as, speakers, lights, and displays.Input subsystem 212 and output subsystem 214, are communicativelycoupled to the processor(s) 204 via the bus 206. In someimplementations, input subsystem 212 includes receivers to receiveposition and/or orientation information. For example, a global positionsystem (GPS) receiver to receive GPS data, two more time signals for thecontrol subsystem 203 to create a position measurement based on data inthe signals, such as, time of flight, signal strength, or other data toeffect a position measurement. Also for example, one or moreaccelerometers can provide inertial or directional data in one, two, orthree axes.

Robot 200 includes a propulsion or motion subsystem 216 comprisingmotors, actuators, drivetrain, wheels, and the like to propel or movethe robot 200 within a physical space and interact with it.

Robot 200 includes a manipulation subsystem 218, for example comprisingone or more arms, end-effectors, associated motors, solenoids, otheractuators, linkages, drive-belts, and the like coupled and operable tocause the arm(s) and/or end-effector(s) to move within a range ofmotions. The manipulation subsystem 218 is communicatively coupled toprocessor(s) 204 via bus 206.

A person of ordinary skill in the art will appreciate the components inrobot 200 may be varied, combined, split, omitted, or the like. In someimplementations one or more of network interface subsystem 210, inputsubsystem 212, output subsystem 214, propulsion or motion subsystem 216and/or manipulation subsystem 218 are combined. In some implementations,one or more of the subsystems (e.g., input subsystem 212) are split intofurther subsystems. In some implementations, the bus 206 is a pluralityof buses (e.g., data buses, instruction buses, power buses) included inat least one body. For example, as part of a modular computingarchitecture where computational resources at distributed over thecomponents of robot 200. That is, a robotic apparatus, like robot 200,could in one embodiment have a processor in a left arm and a storagedevice in its thorax. In some implementations, computational resourcesare located in the interstitial spaces between structural or mechanicalcomponents of robot 200. A data storage device could be in a leg and aseparate data storage device in another limb. In some implementations,the computational resources distributed over the body include redundantcomputational resources.

The at least one processor 204 may be any logic processing unit, such asone or more microprocessors, central processing units (CPUs), digitalsignal processors (DSPs), graphics processing units (GPUs),application-specific integrated circuits (ASICs), programmable gatearrays (PGAs), programmed logic units (PLUs), and the like. Theprocessor 204 may be referred to in the singular, but may be two or moreprocessors.

Network interface subsystem 210 may be any circuitry effectingbidirectional communication of processor-readable data, andprocessor-executable instructions. Suitable communication protocolsinclude FTP, HTTP, Web Services, SOAP with XML, and the like.

The at least one storage device 208 is at least one nontransitory ortangible storage device. In some implementations, storage device 208includes two or more distinct devices. The storage device(s) 208 can,for example, include one or more a volatile storage devices, forinstance random access memory (RAM), and one or more non-volatilestorage devices, for instance read only memory (ROM), Flash memory,magnetic hard disk (HDD), optical disk, solid state disk (SSD), and thelike. A person of skill in the art will appreciate storage may beimplemented in a variety of ways such as a read only memory (ROM),random access memory (RAM), a hard disk drive (HDD), a network drive,flash memory, digital versatile disk (DVD), any other forms ofcomputer-readable memory or storage medium, and/or a combinationthereof. Storage can be read only or read-write as needed. Further,modern computer systems and techniques conflate volatile storage andnon-volatile storage, for example, caching, using solid-state devices ashard drives, in-memory data processing, and the like.

The at least one storage device 208 includes or storesprocessor-executable instructions and/or processor-readable data 250associated with the operation of robot 200, system 100, and the like.

The execution of the processor-executable instructions and/or data causethe at least one processor 204 to carry out various methods and actions,for example via the propulsion or motion subsystem 216 and/ormanipulation subsystem 218. The processor(s) 204 can cause a roboticapparatus, such as robot 200, to carry out various methods and actions.Processor-executable instructions and/or data 250 can, for example,include a basic input/output system (BIOS) 252, an operating system 254,drivers 256, communication instructions and data 258, input instructionsand data 260, output instructions and data 262, motion instructions anddata 264, and executive instructions and data 266.

An exemplary operating system 254 includes ANDROID™, LINUX®, andWINDOWS®. The drivers 256 include processor-executable instructions anddata that allow processor(s) 204 to control circuitry of robot 200. Theprocessor-executable communication instructions and data 258 includeprocessor-executable instructions and data to implement communicationsbetween the robot 200 and an operator console or terminal, a computer,or the like. The processor-executable input instructions or data 260guide robot 200 in processing input from sensors in input subsystem 212.Processor-executable output instructions or data 262 guide the robot 200in interacting within the environment via components of manipulationsubsystem 218 or output subsystem 214. Processor-executable motioninstructions and data 264 guide robot 200 in moving within itsenvironment via components in propulsion or motion subsystem 216.

The processor-executable executive instructions and data 266 guide therobot 200 in reasoning, problem solving, planning tasks, performingtasks, and the like. The processor-executable executive instructions anddata 266 implement, in part, the methods described herein, includingthose in and in relation to FIG. 5, etc.

The processor-executable agent instructions and data 268 guide the robot200 in performing tasks associated with agents, such as, start agent,stop agent, run processor-executable instructions associated with agent,send data to or from agent, and receive data at agent. Theprocessor-executable agent instructions and data 270 implement, in part,the methods described herein, including those in and in relation toFIGS. 10-13.

The processor-executable artificial neural network instructions and data268 allow the robot to operate and train an artificial neural network.The processor-executable artificial neural network instructions and data270 implement, in part, the methods described herein, including those inand in relation to FIGS. 5-9, 15, and 16.

The processor-executable artificial neural networks instructions anddata 270 when executed by robot 200 allow the robot to simulateartificial neural networks including forward direction or mode, e.g.,feedforward operation, and backward direction, e.g., training anartificial neural network. Examples of methods for operation ofartificial neural networks are illustrated in and described with respectto, at least, FIGS. 5, 6, and 8. As used herein and in the claims, theterm “artificial” refers to hardware circuitry or hardware processorimplementation, i.e., a non-human embodiment, which is distinct from anatural neural network of, for example, the human brain.

Input subsystem 212 comprises sensors or transducers that acquire datafor the robot. The data includes sensor information. Sensor informationincludes environmental sensor information representative ofenvironmental conditions external to robot 200. Sensor informationincludes robotic conditions or state sensor information representativeof conditions or states of the robot including the various subsystemsand components thereof. Such sensors may include one or more of camerasor imagers (e.g., responsive in visible and/or nonvisible ranges of theelectromagnetic spectrum including for instance infrared andultraviolet), radars, sonars, touch sensors, pressure sensors, loadcells, microphones, meteorological sensors, chemical sensors, or thelike. Exemplary sensors include camera 156 and microphone 158. Sensorinformation can, for example, include diagnostic sensor information thatis useful in diagnosing a condition or state of robot 200 or environmentin which the robot 200 operates. For example, such sensors may includecontact sensors, force sensors, strain gages, vibration sensors,position sensors, attitude sensors, accelerometers, and the like. Insome implementations, the diagnostic sensors include sensors to monitora condition and/or health of an on-board power source (e.g., batteryarray, ultra-capacitor array, fuel cell array).

Output subsystem 214 comprises one or more output devices. The outputsubsystem 214 allows robot 200 to send signals into the robot'senvironment. Example output devices are speakers, displays, lights, andthe like. Robot 200 may communicate with an agent, such as, a person,and another robot.

FIG. 3 illustrates an exemplary robot 300. As discussed herein, robotsmay take any of a wide variety of forms. These include human operatorcontrollable robots, autonomous robots with on-board control, autonomousrobots controlled by a non-human operator, and hybrid robot (i.e.,partially autonomous, partially piloted). A robot comprises one or morebodies, also called structural components, or brackets. The bodies arecoupled by joints, for example, bearings, and/or servo-motors. Forexample, a first body is connected to a second body by a servo or thelike. It is possible to describe a robot in terms of the joints or thebodies. FIG. 3 is described in terms of the joints but a person of skillin the art will appreciate a body based description is possible.

In various implementations, shoulder servos 302 and 305 may control andsense roll and pitch respectively of a shoulder of a first arm 301 ofrobot 300. In some implementations, shoulder servos 302 and 305 may, forexample be DYNAMIXEL™ MX-28, MX-64, or AX-12 servo-motors produced byROBOTIS CO. LTD. of Seoul, South Korea.

In some implementations, the shoulder yaw servo 304 may control andsense the yaw of first arm 301 of the robot 300. In variousimplementations, the shoulder yaw servo 304 may be a servo-motor likeshoulder servos 302 and 305. Yaw is a motion analogous to medialrotation (i.e., inward rotation toward the body) and lateral rotation(i.e., outward rotation away from the body).

In some implementations, elbow servo 306 may control and sense an elbowof the first arm 301 of robot 300. In various implementations, the elbowservo 306 may be a servo-motor like shoulder servos 302 and 305.

In some implementations, the wrist servo 308 may control and sense anend-effector rotation of the robot 300. In some implementations, thewrist servo 308 maybe a servo-motor as described herein and includingservos for shoulder servos 302 and 305.

In various implementations, the end-effector 310 may include a pluralityof digits 317. For example, four fingers and a thumb are shown in FIG.3. A thumb is generally regarded as a digit that may be used to opposetwo more digits. In the case of an opposed pair of digits the thumb maybe the shorter or less mobile digit. In some implementations, the digitsof the end-effector 310 may include embedded force sensitive resistors.Respective servos, which may, for example be DYNAMIXEL™ XL-320servo-motors or TOWERPRO™ hobby servos, may operate each digitindependently. The end-effectors may, in some implementations,facilitate dexterous manipulation of objects. Each of shoulder servos302 and 305, and servo in robot 300, work cooperatively with arespective joint, or joint and gearbox. In various implementations rollis adduction (i.e., appendage moves toward torso) and abduction (i.e.,appendage moves away from torso) of first arm 301. In variousimplementations pitch is flexion (i.e., appendage reduces angle betweenitself torso of more proximal appendage) and extension (i.e., appendageincreases angle) (e.g., backward) of first arm 301.

In some implementations, one or more digits of digits 317 of theend-effector 310 may have polymer filled internal and external structureand/or rubber pads proximate to the extremities of the one or moredigits of digits 317. The material may, in operation enhance gripcapacity of an end-effector and simulate the resistance of a humanfinger.

In some implementations, digits, such as digits 317, may each have oneor more contact sensors and/or pressure sensors to sense pressureapplied to the sensor and produce signals proportional to the pressure.

The second arm 303 is generally similar to the first arm 301 butmirrored. Referring to FIG. 3, the second arm 303 includes a shoulderroll servo 352, a shoulder pitch servo 315, a shoulder yaw servo 313, anelbow servo 307, a wrist servo 309, and end-effector 311 including aplurality of digits 356.

In at least one implementation, the robot 300 includes one or morecomponents including wheels, such as wheel 312 and wheel 352, anelectronics compartment 316, DC motors 318-1 and 318-2, a speaker 320, awaist pitch servo(s) 322, an interlock 326 (to share torso support withwaist pitch servo(s) 322), a single board computer (SBC) (not shown),two neck servos including a head pitch servo 330 and a head yaw servo331, ear servos 334 and 335, cameras 336 and 337, microphones 338 and339, lights/LEDs 354 and cable bundles (not shown).

In some implementations, wheel 312 and wheel 352 provide the capacityfor locomotion to the robot 300. Wheel 312 and wheel 352 may provide abroad base which, in some examples, increases stability of the robot300. In other implementations, one or more treads can providelocomotion.

In various implementations for example, one or more on-board powersources may be found in the compartment 316. The on-board power sourcescan, for example include one or more batteries, ultra-capacitors, fuelcells, to independently power different components of the robot 300.Servos can be on divided up over different batteries.

Exemplary batteries include secondary cells, for instance lithiumpolymer cells, for example, a 16V, 10000 mAh, four cell, LiPo battery; a4000 mAh 3 cell 12 V battery; a 5 V 9600 mAh, USB mobile charging powerpack; and a batter pack including one or more 3.7 V lithium ionbatteries.

FIG. 4 schematically shows parts of a computer system 400, including aprocessor, for use as a host in the system 100, shown in FIG. 1 inaccordance with the present system, devices, articles, and methods.System 400 shares some similar components with robot 200 but differs inlacking the propulsion or motion sub-system and the manipulationsub-system. System 400 has different sub-components within somesub-systems, such as, the input and output sub-systems.

System 400 includes at least one body or housing 402, and a controlsubsystem 403 that includes at least one processor 404, at least onenontransitory computer- or processor-readable storage device 408, and atleast one bus 406 to which the at least one processor 404 and the atleast one nontransitory computer- or processor-readable storage device408 are communicatively coupled. At least one processor 404 may be anylogic processing unit, such as one or more microprocessors, centralprocessing units (CPUs), digital signal processors (DSPs), graphicsprocessing units (GPUs), application-specific integrated circuits(ASICs), programmable gate arrays (PGAs), programmed logic units (PLUs),and the like. Processor(s) 404 may be referred to in the singular, butmay be two or more processors.

System 400 includes a network interface subsystem 410 is communicativelycoupled to bus 406 and provides bi-directional communicative coupling toother systems (e.g., a system external to computer system 400) vianetwork or non-network communications channel(s) (e.g., communicationchannel(s) 108). Network interface subsystem 210 includes circuitry.Network interface subsystem 210 may use a communication protocols (e.g.,FTP, HTTP, Web Services, and SOAP with XML) to effect bidirectionalcommunication of information including processor-readable data, andprocessor-executable instructions.

System 400 includes an input subsystem 412. In some implementations,input subsystem 412 includes one or more user interface input devicessuch as a display a keyboard, a mouse, a microphone, and a camera. Insome implementations, the input subsystem 412 includes one or moresensors such as environmental sensors. In some implementations, inputsubsystem 412 is coupled to the control subsystem 403 via the networkinterface subsystem 410. The system 400 includes an output subsystem 414comprising one or more output devices, such as, displays, speakers, andlights. Bus 406 brings input subsystem 412, output subsystem 414, andprocessor(s) 404 in communication with each other. The at least onenontransitory computer- or processor-readable storage device 408includes at least one nontransitory storage medium. In someimplementations, storage device(s) 408 includes two or more distinctdevices. Storage device(s) 408 can, for example, include one or more avolatile storage devices, for instance random access memory (RAM), andone or more non-volatile storage devices, for instance read only memory(ROM), Flash memory, magnetic hard disk (HDD), optical disk, solid statedisk (SSD), and the like. A person of skill in the art will appreciatestorage may be implemented in a variety of ways such as a read onlymemory (ROM), random access memory (RAM), a hard disk drive (HDD), anetwork drive, flash memory, digital versatile disk (DVD), any otherforms of computer-readable memory or storage medium, and/or acombination thereof. Storage can be read only or read-write as needed.Further, modern computer systems and techniques conflate volatilestorage and non-volatile storage, for example, caching, usingsolid-state devices as hard drives, in-memory data processing, and thelike. The at least one storage device 208 may store on or within theincluded storage media processor-readable data, and/orprocessor-executable instructions.

The storage device(s) 408 includes or stores processor-executableinstructions and/or data 450 associated with the operation of system400. In some implementations, the processor-executable instructionsand/or data 450 includes a BIOS 252, an operating system 254, drivers256, communication instructions and data 258, input instructions anddata 260, output instructions and data 262, agent instructions and data268, artificial neural network instructions and data 270, and systemcoordination instructions and data 472. The processor-executableinstructions and/or data 252-262, 268, and 270 are described herein andwith appropriate changes are applicable to system 400.

The processor-executable system coordination instructions and data 472guide the system 400 to start, run, and stop one or more agents. Theinstructions and data 472 guide the system in establishing andmaintaining communication between agents. The instructions and data 472when executed may direct the system to record data generated in theoperation of a system of distributed agents. The processor-executableagent instructions and data 268, processor-executable task instructionsand data 270, and/or processor-executable system coordinationinstructions and data 472, implement, in part, the methods describedherein, including those in and in relation to FIGS. 10-13.

FIG. 5 shows a method 500 executable by circuitry or at least onehardware processor, that implements various techniques to operate andtrain at least one artificial neural network (ANN). Those of skill inthe art will appreciate that other acts may be included, removed, and/orvaried to accommodate alternative implementations. Method 500 isdescribed as being performed by a controller, for example, system 400,robot 102 or computer 106, including a control subsystem orprocessor(s). However, method 500 may be performed by another system orcircuitry. For example, method 500 can be performed by in part by anon-board control subsystem (i.e., on board the robot) 203 inconjunction, or cooperation, with an off-board control system (i.e.,external or distinctly physically separate from the robot).

Method 500 begins, for example, in response to an invocation by thecontroller. At 502, the controller sets-up and directs a computingsystem to simulate an artificial neural network. That is, run or operatethe ANN in a forward direction or a backward direction. The controllersimulates the ANN by executing processor-executable instructions thatdefine the ANN. Examples of a suitable computing system include thesystem 100 (FIG. 1). Examples of artificial neural networks includefeedforward neural networks, and recurrent neural networks. In someimplementations, the controller directs a distributed system or runs orsimulates the artificial neural network.

At 504, the controller simulates the operations of the artificial neuralnetwork in a forward direction, phase, or mode. The simulation of theartificial neural network includes modeling the response of theartificial neural network to input. The response of the neural networkto input is, for some networks, a computation. In some implementations,the controller stores data created in by the simulation of the ANN inthe forward direction. The controller may use the artificial neuralnetwork to effect control of a robot. That is, direct a robot's motions.In some implementations, the controller directs a distributed system tosimulate the artificial neural network in the forward direction. In someimplementations, the controller interleaves act 504 with act 506.

An exemplary artificial neural network includes a plurality of units anda plurality of edges. A processor, or other circuitry such as,artificial neural network circuitry, implement a unit by executingprocessor-executable instructions and data. A unit accepts one or moreinputs and applies a non-linear transform to this input. The output ofthe non-linear transform, or activation function, is both the state ofthe unit and the output of the unit and is recorded, or read from,computer-readable media including processor-readable data. The units ina network may be arranged into groupings of one or more units. Aconventional grouping is a layer that includes units of same depth inthe network relative to other units and/or layers. Depth may beexpressed as a count of links. Sometimes layers are characterized by anabsence of intra-layer edges. The output and state of a unit is a linearcombination of inputs, e.g., previous layer, transformed by anactivation function. This maybe expressed as:

$\begin{matrix}{a_{j} = {g\left( {\sum_{i = 0}^{n}{W_{i,j}a_{i}}} \right)}} & (1)\end{matrix}$

The activation a_(i) is for unit i, W_(i,j) is a matrix including theweights between the input units to unit j, and g is an activationfunction. The matrix W_(i,j) is recorded in, and read from,computer-readable media including processor-readable data. Theactivation is a state of a unit and a number represents the activation.In some implementations, the activation of a unit is a value betweenzero and one. An activation function, such as g, has two useful aspects.One, an activation function is non-linear in effect. If the activationfunctions were linear any multi-layer ANN could collapse (e.g.,rewritten) to a single input and output layer. Examples of non-linearactivation functions include non-linear functions, piecewise linearfunctions, and step functions. Two, an activation function provides anoutput regarded as active, for example, a value of one or almost one,for the correct input and provides an output regarded as inactive, forexample, a value of zero or almost zero, for the incorrect input. Insome implementations, correct and incorrect are separated by a thresholdvalue. One suitable function is proportional to a sigmoid function.Alternatives suitable functions are proportional to symmetric sigmoidfunction, softmax function, Heavyside function, complementary bipolarfunction, hyperbolic tangent, LeCun's hyperbolic tangent function, logitfunction, or a product of a polynomial function and any of the forgoing.

At 506, controller trains the artificial neural network over data. Thisis termed as operating in the backward direction. In someimplementations, the data comes from other hosts or robots. In someimplementations, the data is from one or more environmental sensors toone or more robots. Training an artificial neural network involvessimulating the network and evolving (or adapting) of weights for theunits and edges of the neural network until an exit condition isreached. Examples of exit or termination conditions include minimumvalue of objective function, value of objective function below athreshold, lack of change in an objective function, time, and number ofsteps. The weights for the units and edge are parameters of the ANN andused in the forward direction. In some implementations, a gradientdecent based method is used. In some implementations, a backpropagationmethod is used. In some implementations, the controller determines aplurality of incremental errors associated with the variables in theANN. In some implementations, the controller stores data created duringthe simulation of the ANN in the backward direction.

In some implementations, act 506 and act 508 are interleaved. Theinterleaved acts overlap in time for part of the duration of each act.One act can begin before another; end after the other; begin and endduring the other's duration; begin, suspend, and resume while the otheris doing much the same; and the like.

At 508, the controller tests if method 500 is to loop, e.g. is atermination condition present. If the method 500 is to loop, (e.g., notermination condition present, i.e., 508—Yes) control passes to act 504.If the method is not to loop, (e.g., a termination condition is present,i.e., 508—No) the method 500 ends until invoked again.

At optional act 510, the controller returns the weights and otherinformation about the network and the training. “About” is employed herein the sense of represent, characterize, or summarize. For example, thecontroller returns information characterizing or representing the numberof iterations taken in a training method, the nature of the trainingmethod, or hyper-parameters used in the training method.

A recurrent network is an artificial neural network including cycles.Recurrent networks include special hidden units (sometimes calledcontext units), that are determined not just by time, t, but by theinternal state of the network at a previous time. Mathematically arecurrent neural network can be an expression for output units o, hiddenunits h, input units i, and context units c (being a special case ofhidden units). A descriptive set of equations includes:

$\begin{matrix}{{{h(t)} = {f_{1}\left\lbrack {{i(t)},{c\left( {t - 1} \right)}} \right\rbrack}}{{o(t)} = {f_{2}\left\lbrack {h(t)} \right\rbrack}}{{c(t)} = {f_{3}\left\lbrack {h(t)} \right\rbrack}}} & (2)\end{matrix}$

The hidden units are a function of input, and previous state of thecontext units. The output units and the context units are a function ofhidden units. Here, time is a natural number. Continuous time recurrentnetworks are different in that time, t, is a value on the real numberline. A system of differential equations is one way to expresscontinuous-time dynamics for a network. An example is a differentialequation in the rate of change of activation at a hidden unit. Anotheris a system of differential and algebraic equations. For example:

$\begin{matrix}{{h^{\prime{(t)}} = {{\alpha{h(t)}} + {\beta{f_{1}\left\lbrack {{i(t)},{c\left( {t - 1} \right)}} \right\rbrack}}}}{{o(t)} = {f_{2}\left\lbrack {h(t)} \right\rbrack}}{{c(t)} = {f_{3}\left\lbrack {h(t)} \right\rbrack}}} & (3)\end{matrix}$

Where α and β are values that need to be learned such that the recurrentnetwork can be used to process information from sensor(s), e.g.,real-time streams of stimuli. The outputs of recurrent networks may becontrol signals to make a robot, move, talk, and the like; or be motion,speech, and the like.

Revisiting FIG. 5, the illustrated method includes a loop of includingrunning an ANN in a forward direction and a backward direction. Focusingon acts 504 and 506, there is a method that begins at invocation by thecontroller. The controller runs the artificial neural network in theforward direction. The controller may effect control of a robot overtime. The controller operates the artificial neural network in thebackward direction. That is, updates the parameters of the ANN in viewof the input, output, and objective function. In some implementations,the controller uses back propagation through time. In someimplementations, the controller determines a gradient for the ANN andthen updates the parameters. In some implementations, the controlleruses a gradient descent, e.g., standard, stochastic, and momentumenhanced.

In some embodiments, the ANN is a recurrent neural network. A recurrentneural network includes cycles of edges. In some embodiments, thecontroller runs the recurrent neural network backward by first unfoldingthe recurrent neural network. Then the controller backpropagates tooptimize an objective function over the unfolded recurrent neuralnetwork

In some embodiments, the controller operates the artificial neuralnetwork in the forward direction. The controller then operates theartificial neural network in the backward direction. The time thecontroller operates the ANN forward direction can be called a forwardphase. A backward phase is counterpart to the forward phase. In someimplementations, there are a plurality of different types of forwarddirection, phase, or mode. The controller continues the alternation offorward and backward phases until termination described herein.

The forward and backward operations are sequential and alternating. Insome implementations, the operations are sequential but not strictlyalternating. In some implementations, the operations are performed inparallel.

In some embodiments, the controller simulates the ANN in the forwarddirection by numerically integrating differential equations thatdescribe the operation of the ANN. For example, where the ANN is arecurrent neural network and modelled as a continuous time recurrentneural network then a first order differential equation in the rate ofchange of activation at a unit can be integrated to determine theactivation at the unit. A differential equation is often expressed inexplicit form as derivative of a given order for a dependent variableequal to a function of an independent variable, the dependent variable,and one or more derivatives of the dependent variable of an order lessthan the given order. For example, y′=f(x,y). One can find a solution tothe differential equation by discretizing the variables and numericallyintegrating the discretized equation. Example integration techniquesinclude determining a sequence of values according to:

$\begin{matrix}{y_{n + 1} = {y_{n} + {\left( {1 - \theta} \right){{hf}\left( {x_{n},y_{n},y_{n}^{(1)},\ldots}\; \right)}} + {\theta\; h\;{f\left( {x_{n + 1},y_{n + 1},y_{n + 1}^{(1)},\ldots}\; \right)}}}} & (4)\end{matrix}$

Where y_(n) is the dependent variable at step n, x_(n) is the dependentvariable at the same step, h is the different between step n+1 and n, fis the differential equation in explicit form, and θ is a tunableparameter. For example, function f can be the rate of change ofactivation at a unit in an artificial neural network. When θ=0 thetechnique is Forward Euler. When θ=1 it is Backward Euler. Intermediatevalues are hybrid techniques and θ=½ corresponds to the Crank-Nicolsonmethod. Backward Euler and hybrid techniques are more numericallystable.

FIG. 6A and FIG. 6B are schematic diagrams illustrating a portion of anartificial neural network (ANN) 600 including a plurality of inputs 602,a plurality of edges 603, a plurality of sets of ANN variables 604, anda plurality of outputs 606. ANN 600 includes a plurality of edges 603disposed between the plurality of inputs 602, the plurality of ANNvariables 604, and the plurality of outputs 606. In someimplementations, one or more sets of ANN variables in the plurality ofANN variables 604 includes ANN variables and further edges. FIGS. 6A and6B depict the flow of information through the ANN 600. As illustrated,the information is shown flowing in the forward direction along edges603.

In some implementations, the ANN 600 handles multimodal inputs. Forexample, the ANN 600 receives input, as sensor information, from one ormore sensors, either directly or indirectly, for instance from one ormore imagers (e.g., video, sequential images), microphones (e.g.,audio), motor sensors or motors (e.g., rotational encoder, steppermotor), touch sensors, force sensor, and the like. The ANN receives someinput at regular intervals, e.g., video at 30 frames per second andaudio. Some input is received episodically, e.g., motor information whenmotors are running. Some input arrives irregularly, or in an eventdriven way, e.g. speech word-recognition.

In some implementations, the ANN 600 has output. For example, the ANN600 emits plurality of outputs 606 of one or more types. Examples oftypes of output include motion, actuator output, speech, audio, video,and the like. The rate of output can differ between outputs in theplurality of outputs 606.

In some implementations, the ANN 600 runs in real time. The ANN 600 runsacross a plurality of hosts (e.g., computers, robots). In someimplementations, the ANN runs distributed on a plurality of hosts with avariety computational capacities.

FIG. 6B illustrates the portion of ANN 600 distributed over two hosts.In the illustrated example, the ANN is simulated on two hosts, i.e.,host 652 and host 654. A portion of the plurality of edges couple partsof ANN 600 in one host to parts of ANN 600 in another host.

Operation of an ANN on a distributed plurality of hosts is, withtechnical and practical effect, different from a non-distributed system.The information flow between the hosts has two directions. The hosts mayoperate in a forward direction (as shown) with information flowing frominput layer of ANN to output layer and routed between hosts as may beneeded. The hosts may operate in a backward direction with the reverseinformation flow. For example, in forward direction a first host, e.g.,host 652, creates output consumed by other hosts, e.g., host 654. Inforward direction the one host is a source, the other hosts are sinksarranged for instance in a fan-out structure. In backward operation, therole of sources and sinks are reversed with one host assuming a collectroll. However, fan-out and collect operations don't paralyze in the sameway. For example, every “fire off” and “forget forward” message in afan-out operation is a potential “check, wait, recheck” (e.g., spinlock) backward message in a collect operation. Thus an ANN distributedover a plurality of hosts will need to, for either the forward orbackward operations, handle delays in receiving variables due tocommunication lag and computation lag.

The operation of an ANN on a distributed plurality of sets of ANNvariables 604 is technically different from a non-distributed pluralityof sets of ANN variables. The operation of an ANN in a distributedenvironment also leads to spin locking as described herein above.

The execution of a distributed application, characterized by a directedacyclic graph, by two more hosts is technically different from anon-distributed execution of the same application by one host. Thedependencies of the parts of the application have to be accounted for.As well, if the directed acyclic graph is not distributed along with theparts of the application then inverting the application is hard for anyhost.

System 600 can include two or more artificial neural networks. In someimplementations, one or more hosts simulate two or more artificialneural networks. The two or more artificial neural networks can havedifferent objective functions. One network could implement a controlsystem for a robot. When the ANN is used as a control system to controla robot, an example objective function is a suitable reward function tooptimize control of the robot. An example of an objective functionincludes a task specific cost function, an error function parameterizedby demonstrated motion, or a value function. In some implementations, anobjective function is proportional the absolute difference between a setof target values and the output of the ANN. The set of target values maybe derived from information characterizing or representing demonstratedmotion of the robot via information received from an operator interface,such as, operator interface 104.

A second network, within system 600 or similar system, could implement apredictive control model. A second network, within system 600 or similarsystem, could implement a system model. An example of an objectivefunction may vary with the application for the model. When the ANN isused to create and store a model of the environment, a suitableobjective function is proportional the absolute difference between a setof target values and the output of the ANN. The set of target values canbe drawn from information characterizing or representing an environment,such as, environment to a host. That is, an objective function based ondifferences between the system model for an environment and the actualor real environment.

FIG. 7 shows a method 700 operation in forward mode and backward mode insequential acts. Method 700 does not have a two-part alternation likethat shown in FIG. 5, but rather a more complicated cadence with unequaldurations of phase. Those of skill in the art will appreciate that otheracts in method 700 may be included, some acts omitted, and/or actsperformed in different orders to accommodate alternativeimplementations. The method 700 is described as being performed by acontroller, for example, robot 102 or computer 106, including a controlsubsystem or processor(s). However, the method 700 may be performed byanother system, and more than one computing system. As well, the method700 can be performed off-line to train an ANN.

The method 700 is potentially part of a larger method. That is, method700 begins at 702 or at a time prior to act 702. For example, method 700began many episodes previously to present time.

At 702, the controller simulates an ANN in the forward direction. Thecontroller calculates one or more sets of values for the variablespresent in the ANN. The controller simulates the ANN until informed thatthe present period or episode has expired. An example episode length isseveral seconds. In some implementations, time is discretized and anepisode is a number of time steps. That is time is a value on thenatural number line. During the forward direction simulation 702, thecontroller stores values generated in the simulation of the ANN.

At 704, the controller receives information characterizing orrepresenting notice of the end of a present episode. For example,episodes can be numbered and the present episode referred to as N. Insome implementations, the information characterizes or represents thetime bonds on the episode: start time and end time. The end time neednot be coterminous with the present time in act 704. That is the forwardoperation can occur without knowledge of episode information.

At 706, the controller simulates the ANN in the backward direction. Thatis, the controller trains the ANN based on values stored in act 702. Thecontroller steps backward through time in the present episode (i.e., theepisode just completed). In some implementations, the controllerdetermines new parameters, e.g., weights, for the ANN at each time step.In some implementations, the controller determines new parameters forthe ANN at the “start” of the present episode. That is, the end of theepisode since act 706 runs in reverse over the episode. In someimplementations, the controller calculates gradients for an objectivefunction with respect to the parameters of the ANN. In someimplementations, the controller uses the gradients to update theparameters using gradient descent including the versions of gradientdescent described herein.

At 708 the controller updates information characterizing or representingthe parameters within the artificial neural network. For example, thecontroller makes available the parameters for future forward directionoperations. In some implementations, the controller distributes theinformation characterizing or representing the parameters within theartificial neural network, e.g., weights, to other agents in adistributed system.

At 710, the controller simulates an ANN in the forward direction. Act710 and act 702 are, in some implementations, the same or identical inoperation. The controller calculates one or more sets of values for thevariables present in the ANN. The controller continues until receipt ofnotice of the end of the next episode, e.g., N+1. Again, the bounds onthe next episode may unequal to the bounds on act 710. The duration ofthe episode is less than or equal to the duration the controllersimulates the ANN over.

The process continues until termination, executing in a forwarddirection, e.g., act 702, followed backward direction, e.g., acts 706,and then parameter distribution, e.g., act 708. In some implementations,a controller may execute method 700 to create a substantiallyalternating cadence. In some implementations, the duration of thepresent episode and the duration of act 702 are the same in temporallength. In some implementations, the present and next episodes have thesame duration as one another. In some implementations, the method 700has a substantially episode or episodic cadence, e.g., long forwardphases interrupted by other acts. The method 700 is sequential in thatthe backward phase at act 706 may block the operation of a greaterapplication, such as, an application defined by an ANN.

FIG. 8 shows a method 800 including forward and backward phasesinterleaved together. Those of skill in the art will appreciate thatother acts in method 800 may be included, some acts omitted, and/or actsperformed in different orders to accommodate alternativeimplementations. The sequence in FIG. 8 is described as being performedby a controller, for example, robot 102 or computer 106, including acontrol subsystem or processor(s). However, the method 800 may beperformed by another system, and/or by more than one computing system.As well, the method 800 can be performed off-line to train an ANN.

The method 800 begins at 802 or at a time prior to act 802. For example,method 800 may have begun many episodes previously to a present time.Method 800, which uses previous values, can be boot strapped from storedvalues in place of previous values.

At 802, the controller simulates the ANN in the backward direction. Thatis, the controller trains the ANN. The controller steps backward throughtime over a previous episode, e.g., N−1. In some implementation, thecontroller runs the recurrent neural network backward by unfolding theANN over the previous episode. In some implementations, the controllerdetermines new parameters, e.g., weights, for the ANN at each time step.In some implementations, the controller determines new parameters forthe ANN at the “start” of the previous episode. That is, the end of theprevious episode since act 802 runs in reverse over the episode. In someimplementations, the controller calculates gradients for an objectivefunction with respect to the parameters of the ANN. In someimplementations, the controller uses the gradients to update theparameters using gradient descent including the versions of gradientdescent described herein. In some implementations and examples of ANNand computing capacities, act 802 is interleaved with act 804, a forwardphase.

At 804, the controller simulates an ANN in the forward direction. Thecontroller calculates one or more sets of values for the variablespresent in the ANN. The controller uses parameters from an earlier time,such as, two episodes previously, i.e., N−2.

At 806, the controller receives information characterizing orrepresenting notice of the end of a present episode, i.e., N. Theinformation can be received from a central controller or another agent.In some implementations, the information characterizes or represents thetime bounds on the episode, start time and end time. The end time neednot be coterminous with the present time in act 806. That is, theforward operation can occur without knowledge of episode information. Anexample episode length is several seconds. In some implementations, timeis discretized and an episode is a number of time steps.

At 808 the controller updates information characterizing or representingthe previously updated parameters for the network. The previouslyupdated parameters were created in act 802. That is, the controllermakes available the weights for operations, e.g., running in forwarddirection. In some implementations, the controller returns thepreviously updated parameters. In some implementations, the controllerdistributes the previously updated parameters. This is a fasteroperation than generating the parameters. For example, a pointer updatecould effect a fast update of parameters.

At act 810, the controller simulates the ANN in the backward direction.The controller steps backward through time over the “present” episode,e.g., N. Act 810 is substantially the same in operation as act 802, butis executed for the just ended episode N. Act 810, as well, isinterleaved with at least the start of act 812.

At act 812 the controller simulates the ANN in the forward direction.The controller simulates the ANN waiting for a notice indicating the endof the “next” episode, e.g., N+1. The variation stated in relation toact 804 are applicable here. Act 812 overlaps with act 810 for at leastthe start of act 812.

Method 800 continues until termination. The controller runs a backwardphase, e.g. act 802, potentially interleaved forward phase, e.g. act804, then at the end of an episode provides a parameter distribution,e.g., 808. The process continues with backward phase, e.g., act 810 or802, potentially interleaved forward phase, e.g. act 812 or 804, then atthe end of an episode, a parameter distribution, e.g., 808.

FIG. 9 shows a sequence diagram for a method 900 using two buffers.Those of skill in the art will appreciate that other acts may beincluded, some acts omitted, and/or acts performed in different ordersto accommodate alternative implementations. The method 900 is describedas being performed by a controller, for example, robot 102 or computer106, including a control subsystem or processor(s). However, thesequence may be performed by another system. The exemplary sequence ofoperations shown includes two buffers at one host. A person of skill inthe art will appreciate that two or more buffers can be used and/orbuffers on other hosts.

The sequence of acts and interactions in method 900 involves acontroller making use of a first buffer 902 and a second buffer 904. Thebuffers can be labeled left and right, up and down, strange and charm,and the like. The buffers can be on the same host or different hosts.Although two buffers are shown the method is extendable to more than twobuffers.

At 906, the controller operates an ANN in the forward direction. Thecontroller calculates values for the variables in the ANN. These valuesare determined for one or more time steps, e.g., t₀ to t_(n), includedin a first episode.

At the end of the first episode, the controller stores the currentvalues for the variables in the ANN in the second buffer 904 at 908. Insome implementations, the current values for the variables in the ANNare recorded as they are generated in act 906.

At 910, the controller determines gradients to an objective function forthe ANN over the first episode. For example, the controller engages inback propagation through time over the first episode for the ANN in thefirst buffer. The first buffer includes the values for the variables ofthe ANN over the episode.

At 912, the controller simulates the ANN for a second episode and storesvalues for the variables in the second buffer 904. The second episodecan be the same length or duration as the first episode or can be adifferent length or duration.

At 914, the controller updates the parameters of the ANN using thegradients determined in 910. The update can be by gradient descent,stochastic gradient descent, mini-batch gradient descent, or batchgradient descent. Generally, an update of parameters occurs faster thandoes a determining of gradients of a cost function with respect to thesame parameters.

At 916, the controller optionally transfers the variables for the ANN atthe end of the second episode from the second buffer to the firstbuffer. At 918, the controller transfers updated parameters for the ANNto the second buffer 904. The relative orders of operations 916 and 918are arbitrary.

At 920 the controller operates the ANN in the forward direction. Thecontroller calculates values for the variables in the ANN. These valuesare determined for a third episode.

At 922, the controller determines gradients to the objective functionfor the ANN over the second episode. For example, the controller engagesin back propagation through time over the second episode for the ANN inthe second buffer 904. The second buffer includes the values for thevariables of the ANN over the episode.

At 924, the controller updates the parameters of the ANN using thegradients determined in 922.

At 926, the controller optionally transfers values for the variables forthe ANN over the third episode to the second buffer. At 928, thecontroller transfers updated parameters for the ANN to the first buffer902. The relative orders of operations 926 and 928 are arbitrary.

The process continues as needed with further episodes. In someembodiments, the only updates during the forward direction blockingoperation is the parameter update, which takes little time. This allows,for example, the use of the forward direction to control a robot.Further, the transfer of updated parameters is event driven.

FIG. 10 illustrates a method 1000 of operation for a plurality ofagents. Method 1000 implements techniques to establish and coordinate aplurality of agents over a plurality of hosts. Those of skill in the artwill appreciate that other acts may be included, some acts omitted,and/or acts performed in different orders to accommodate alternativeimplementations. Method 1000 is described as being performed by a systemcontroller, for example, robot 102 or computer 106, including a controlsubsystem or processor(s). However, method 1000 may be performed byanother system. For example, method 1000 can be performed by in part byan on-board control subsystem (i.e., on board the robot) 203 inconjunction or cooperation with an off-board control system (i.e.,external or distinctly physically separate from the robot, with orwithout human interaction, intervention or control).

Method 1000 begins by an invocation by the system controller. At 1002,the system controller receives a system specification document. Thesystem specification document includes information characterizing,representing, or specifying the components within a distributedapplication. The system specification document may be in one or moresemi-structured (i.e., self-describing), or structured formats includingJSON, XML, and binary. At 1004, the system controller parses the systemspecification document. Parsing identifies information thatcharacterizes or represents a plurality of agents in a system and one ormore routes (e.g., communications paths) between agents.

The system controller, at 1006, selects one or more hosts to run theplurality of agents. A host is a tangible or virtual computer. Examplesof a tangible computer include robot 200, robot 300, or computer system400. An agent is processor executable-instructions andprocessor-readable data. Agents are part of the system and may bespecified in the system specification document or pre-existing, e.g.,distributed in a library, or loaded on to a host.

At 1008, the system controller starts one or more agents. In someimplementations, the system controller sends information to one or morehosts, the information specifying one or more requests to start aplurality of agents. For example, the system controller requests a hostto start one or more agents associated with the host. In someimplementations, the system controller requests each previously selectedhost start one or more agents. A description of a method for a host tostart and run of an agent is found herein at least in relation to FIG.12. In some implementations, the agents actively report to the systemcontroller the agents have started. In some implementations, the agentspassively report to the system controller that the agents started. Anexample of passive reporting is updating a data structure.

At 1008, the system controller sends one or more hosts suitable parts ofthe system specification document. The suitable parts of the documentinclude information specifying, for example, the trigger expression andthe body expression.

At 1010, the system controller creates one or more routes betweenagents. That is, the system controller establishes or causes to beestablished communication links between the agents on the various hostsin the system.

At 1012, the system controller monitors the behavior of the agentsand/or one or more routes. The monitoring continues until termination,e.g., a termination request for the system arrives. The system mayprocess the termination by the process described in relation to FIG. 11.

FIG. 11 is a flow-diagram showing an implementation of a method 1100 toterminate the operation of distributed agents. Those of skill in the artwill appreciate that other acts may be included, some acts omitted,and/or acts performed in different orders to accommodate alternativeimplementations. Method 1100 is described as being performed by a systemcontroller, for example, robot 102 or computer 106, including a controlsubsystem or processor(s). However, method 1100 may be performed byanother system. For example, method 1100 can be performed by in part byan on-board control subsystem (i.e., on board the robot) 203 inconjunction or cooperation with an off-board control system (i.e.,external or distinctly physically separate from the robot, with orwithout human interaction, intervention or control).

Method 1100 begins at 1102 with the system controller receiving atermination request for one or more agents in a system. The terminationrequest includes information specifying one or more agents in thesystem. In some examples, the termination request specifies all agentsin the system. At 1104, the system controller stops the routes to andfrom the specified agents. At 1106, the system controller stops thespecified agents. Optionally at 1108, the system controller stores thestates of the specified agents as needed. That is, the system controllerdirects the hosts to store the state of the specified agents, or systemcontroller stores the state of the agents.

FIG. 12 is a flow-diagram of a method 1200 of operation for an agent ina distributed system. The method includes a plurality of acts. Those ofskill in the art will appreciate that other acts may be included, someacts omitted, and/or acts performed in different orders to accommodatealternative implementations. Method 1200 is described as being performedby a host, for example, robot 102 or computer 106, including a controlsubsystem or processor(s). However, method 1200 may be performed byanother system. For example, method 1200 can be performed by in part byan on-board control subsystem (i.e., on board the robot) 203 inconjunction or cooperation with an off-board control system (i.e.,external or distinctly physically separate from the robot, with orwithout human interaction, intervention or control). The description ofmethod 1200 includes setting up one agent, but the process is anapplicable to a plurality of agents.

Method 1200 begins at 1202 with the host receiving a request to start anagent. At 1204, the host receives a part of a system specificationdocument. The part of the system specification includes informationdefining the agent and one or more associated routes. In variousimplementations, routes are set up before agents. In someimplementations, the host optimizes the part of the systemspecification.

In some implementations, an ANN comprises agents. An agent comprisesvarious components including input interfaces, output interfaces (e.g.,queue objects), storage spaces for maintaining an internal state (e.g.,state object), and processor-executable instructions defining executablecomputational logic (e.g., handler objects). In some implementations, anapplication comprises agents.

At 1206, the host opens up the one or more routes. In someimplementations, the host establishes or causes to be establishedcommunication between itself and another host. In some implementations,the host establishes or causes to be established communication between afirst and a second agent. For example between an agent defined in thepart of the system specification and a remote agent.

At 1208, the host creates one or more objects for one or more of queues,state variables, or timers, as defined for the agent. Each queue isassociated with a route into or out of the agent. The queue couples thestate variables to the route. The state variables are internal variablesto the agent. The timers are used to execute processor-readableinstructions in the agent in the absence of an incoming message on aroute or an addition a queue. A timer allows agents to runprocessor-executable instructions on a scheduled basis.

At 1210, the host creates handlers, or handler objects, according to thesystem specification document. A handler may consume the content of aqueue, perform some processing, update one or more states, and/or extendanother queue. In some implementations, the handler includes a triggerexpression and a body expression. An expression is a section ofprocessor-executable instructions. A trigger expression defines orspecifies when a processor or other circuitry should execute a bodyexpression. The body expression contains processor-executableinstructions defining the function of the agent. For example,processor-executable instructions to consume the content of a queue,perform some processing, update one or more states, and/or extendanother queue.

At 1212, the host places itself in a loop. The host loops over all theagents it has defined, e.g., states within the agents, and triggerexpressions. The host loops over all the routes moving data. The hosttests for termination of one or more agents. This loop is describedherein at least in relation to FIG. 13.

FIG. 13 is illustrates an example of a method 1300 of operation for adistributed agent. The method includes a plurality of acts. Those ofskill in the art will appreciate that other acts may be included, someacts omitted, and/or some acts performed in different orders toaccommodate alternative implementations. Method 1300 is described asbeing performed by a host, for example, robot 102 or computer 106,including a control subsystem or processor(s). However, method 1300 maybe performed by another system.

Method 1300 implements techniques to operate an agent. Method 1300begins by an invocation by the host. At 1302, the host starts the agent.The host executes processor-executable instructions to start theoperation of one or more components. The controller maintains the inputinterface and the output interface for the component.

At 1304, the host executes the trigger expression. The host compares ortests for the presence of a trigger condition or activation condition.Examples of a trigger condition include an event, a time, and functionsbased on event or time. For example, is the current time as provided bya timer object equal or more than a predetermined threshold value. Alsofor example, the host determines if there is information characterizingor representing occurrence of an event in a physical environment, withinthe system, or within the host. If there is a trigger condition (i.e.,1304—Yes), control passes to 1306. If there is no trigger condition(i.e., 1304—No), control passes to 1308.

At 1306, the host executes the processor-executable instructions thatdefine computational logic for the component. That is, the host executesthe body expression. Examples of logic within a body expression includecompute a sine wave, compare a set of predicted values to a set oftarget values, and compute an activation function on a weightedsummation of values.

At 1308, the host tests for the presence of a loop terminationcondition. If the loop is to continue (i.e., 1108—Yes), control passesto act 1304. A delay (not shown) can occur before the controllerexecutes instructions that define act 1304. If a termination conditionis present and the loop is to end (i.e., 1308—No), control passes to act1310. At act 1310, the host stops the agent. An example of stopping anagent is described herein at least in relation to FIG. 11.

FIG. 14 is a flow-diagram of a method 1400 of operation a distributedsystem running an application including two more agents. The method 1400includes a plurality of acts. Those of skill in the art will appreciatethat other acts may be included, some acts omitted, and/or some actsperformed in different orders to accommodate alternativeimplementations. Method 1300 is described as being performed by a host,for example, robot 102 or computer 106, including a control subsystem orprocessor(s). However, method 1400 may be performed by another system.For example, method 1400 can be performed by in part by a system likesystem 400 or a control system like subsystem 203.

Method 1400 is suitable for an event driven application. In an eventdriven application, the instance of the ANN varies between executions.To run an ANN in the backward direction a record of a cascade of agentsis needed. In distributed system, there may be no storage space to shareinformation characterizing or representing the cascade of agents. Thus,method 1400 can be used to recover information characterizing orrepresenting the cascade of agents.

Method 1400 is suitable for an application in a system without acorrectness guarantee. Method 1400 is suitable for a system with alatency guarantee (e.g., maximum latency guarantee that system latencywill not exceed a defined amount). The present disclosure includesdescriptions of an ANN operating with a maximum latency guarantee forcommunication on the edges in the ANN. Communication can either operatewith correctness guarantees or maximum latency guarantees, and not both.Artificial neural networks are designed for correctness guarantees. Thatis, ANN are assumed to operate, as far as Applicants are aware, withfaultless edges. Some ANN use dropout techniques to purposely andselectively remove edges, but this is done in training. Whereasreal-time control requires processor-executable instructions designedfor latency guarantees. That is a message will arrive with a certainlatency or it can be assumed that the message will never arrive.

Method 1400 begins at 1402 with the host receiving a part of a systemspecification document. The part of the system specification includesinformation defining the agent and one or more associated routes. At1404, the controller starts two or more agents. The creation and runningof agents is described herein.

In various implementations, the agents communicate with each other in anevent driven way. That is a trigger condition is based on an eventrather than a timer. Each agent includes a set of local connections forthe agent. These connections are found in information specifying theroutes between agents and when the routes are used.

In various implementations, the agents communicate with each other withcommunication delayed due to latency. That is, a trigger is delayedbecause a message is late, but does arrive. The connections betweenagents (i.e., routes) include a temporal component, i.e., when a routeis used, that varies with example agents, routes, activation conditions,and computing resources. Again each agent includes a set of localconnections for the agent. These connections are found in informationspecifying the routes between agents and when the routes are used (i.e.,under which specific conditions a particular route is to be used).

At 1406, the host updates a data structure that tracks a history ofagents. In some implementations, the data structure is a directedacyclic graph. The nodes represent agents or times the agents areexecuted. That is, each agent has a temporally ordered set of localconnections.

At 1408, the host receives training data and an objective function. Theobjective function may operate on predicted and actual output values.The objective function, can have as an operand or input parameters of anANN.

At 1410, the host updates parameters associated with agents in view ofthe training data, the objective function, and the history of agents.For example, the host treats the history of agents as an ANN andoptimizes the objection function by varying parameter in the agents.

At 1412, the host stores or returns the parameters associated withagents.

In some implementations, an agent is implemented using a decoratordesign pattern. A decorator pattern is used to enhance or modify objectsfor a particular purpose. An object the agent can include may take theform of base functionality, like decrementing queues or changing state.The trigger and body expressions can be implemented as a decorator ontop of the agent object.

FIG. 15 is a flow-diagram showing an implementation of a method 1500 forsystem identification. Those of skill in the art will appreciate thatother acts may be included, some acts omitted, and/or acts performed indifferent orders to accommodate alternative implementations. Method 1500is described as being performed by a host, for example, robot 102 orcomputer 106, system 400, including a control subsystem or processor(s).However, method 1500 may be performed by another system.

In some embodiments, the host includes processor-executable instructionsthat define at least two agents. A first agent is a write agent thatwrites input to a model when the trigger condition is satisfied. Themodel's output is a representation of a tangible system in anenvironment, for example, robot 102-1 in environment 140-1. For example,the model's output includes instructions to position the robot or itsend-effector in the environment according to the output. A second agentis a read agent that receives data form a sensor characterizing orrepresenting a state of a tangible system. For example, the data cancharacterize or represent the position of a robot or end-effector.Reading is an acquisition of potentially noisy information thatcharacterizes or represents a state of a robot. Write is an operation toeffect, as in cause, a state. In some implementations, the model is anartificial neural network. The agents can call a third agent thatevolves the state the model.

The agents execute when their respective triggers are satisfied. Forexample, the first agent executes with a first frequency and the secondagent executes with a second frequency. For example, a write agentwrites every 20 milliseconds and a read agent reads every 5milliseconds. In some implementations, the frequency is implemented by atimer within an agent. In some implementations, the first agent or thesecond agent executes in an event driven way. Thus, while the termfrequency is used, operation in at least some instances can beaperiodic.

At 1502, the host's read agent writes input to a model. In someimplementations, the host writes input per a first time series. Forexample, following a sequence of events or at a first frequency. At1504, when the model is an artificial neural networks with hidden units,the host determines the value of the hidden units in view of the input.

At 1506, the host predicts the output of the model for each writeoperation. The output can be generated in an event driven way with newinput. In some implementations, the host causes the robot to take anaction, e.g., move.

At 1508, the host reads actual or true output values. In someimplementations, the host does this according to a second time series.For example, at a second frequency. In some implementations, the readingof true output values occurs at about the same time the host predictsthe output of the model. Here “about” is used in the sense of differingby much less than the period of the faster of the first or secondfrequency, or average frequency of the first or second time series. Forexample, “about” is used in the sense of differing by less than or equalto ten percent of the period of the faster of the first or secondfrequency.

At 1510, the host trains the model in view of the input, output,predicted output, and an error function. An example of incrementalerrors is the difference between output given input and the predictedoutput. For example, the host minimizes an objective function withrespect to parameter of the mode. That is, vary parameters in the modelto minimize the difference between output, and predicted output asmeasured by the objective function. The host then updates the parametersfor the model.

At 1512, the host returns or stores information characterizing orrepresenting the updated parameters. The information can be used toidentify the system. That is, an agent such as a user at an operationinterface like interface 104, will know given the inputs what the systemwill output. For example, the parameters could specify how a servo-motorwill respond to a signal containing desired positions.

FIG. 16A is a flow-diagram showing an implementation of a method 1600for creating a predict model. Those of skill in the art will appreciatethat other acts may be included, some acts omitted, and/or some actsperformed in different orders to accommodate alternativeimplementations. Method 1600 is described as being performed by a host,for example, robot 102 or computer 106, system 400, including a controlsubsystem or processor(s). However, method 1600 may be performed byanother system.

In some embodiments, the host includes processor-executable instructionsand/or data that define at least two agents. A first agent is a writeagent that writes input to a model when the trigger condition issatisfied. A second agent is a read agent that reads the output of amodel. The agents can call a third agent that evolves the state themodel. For example, the third agent includes a handler to, whenrequested (i.e., per a trigger condition), propagate values within anartificial neural network, i.e., body expression.

At 1602, the host's write agent writes input to a model over a firsttime series. For example, write at a first frequency. When the model isan artificial neural network with hidden units, at 1604 the hostdetermines the values for the hidden units in view of the input. In someimplementations, the host calculates the value of the hidden units untilthe end of the first time series.

At 1606, the host produces the output of the model. In someimplementations, the host produces output for each execution of thewrite processor-executable instructions and thus the values of thehidden units. In some implementations, the host can produce output overa second time series. In some embodiments, the host produces the outputby causing a robot to take action. For example, where the robot's stateis a linear transform of the hidden units, the host produces the outputby updating the hidden units. In some embodiments, the host produces theoutput by predicting output of the model.

At 1608, the host reads the output values. In some implementations, thisis done over the second time series. In some implementations, the hostreads the output values in an event driven way. In some implementations,the reading of target output values occurs at about the same time as theproduction of the output of the model. About is used here in the senseof approximately equal to, for example, within ten percent.

At 1610, the host evaluates an objective function with arguments drawnfrom the target output values and the output value over the firstperiod. The objective function expresses how close the input achievedthe target output given the parameters of the model.

At 1612, the host varies the input to optimize the objective function.For example, the host finds the input that brings the robot to thetarget state. Thus, method 1600 can be repeated for different targetoutputs.

The host may return or store information characterizing or representingthe updated inputs. The information can be used as a predictive modelfor the system. That is, based on a knowledge of what input to thesystem attains a target position, a trained system model, could providegood guesses or approximations to future output values given a varietyof input values.

FIG. 16B is a flow-diagram showing a worked example 1650 of method 1600of FIG. 16A. At 1652, at time t the host writes input to a model andcomputes the value of the hidden units within the model. At 1654, attime t plus an increment, the input to the model remains unchanged andthe read agent reads the output, o_(T). The host computes the value ofthe hidden units, h; predicted output, o; and, in some implementations,the value of the objective function, Obj. In some implementations, theevaluation of the objective function is batched.

At 1656, at time t plus two increments, the input to the model remainsand the read agent reads the output, o_(T). The host computes the valueof the hidden units, h; predicted output, o; and, in someimplementations, the value of the objective function, Obj.

This incrementally repeats at 1658 and 1660. Then at 1662, the hostrecords the input that minimizes the objective function. The input isassociated with a target values for the output. The evaluation of theobjective function can occur at 1662.

The above description of illustrated examples, implementations, andembodiments, including what is described in the Abstract, is notintended to be exhaustive or to limit the embodiments to the preciseforms disclosed. Although specific examples are described herein forillustrative purposes, various equivalent modifications can be madewithout departing from the spirit and scope of the disclosure, as willbe recognized by those skilled in the relevant art. The teachingsprovided herein of the various embodiments can be applied to manycomputer systems, robotic systems, and robots, not necessarily theexemplary computer systems, robotic systems, and robots herein andgenerally described above.

For instance, the foregoing detailed description has set forth variousembodiments of the devices and/or processes via the use of blockdiagrams, schematics, and examples. Insofar as such block diagrams,schematics, and examples contain one or more functions and/oroperations, it will be understood by those skilled in the art that eachact and/or operation within such block diagrams, flowcharts, or examplescan be implemented, individually and/or collectively, by a wide range ofhardware, software, firmware, or virtually any combination thereof. Insome embodiments, the present subject matter is implemented viaApplication Specific Integrated Circuits (ASICs). However, those skilledin the art will recognize that the embodiments disclosed herein, inwhole or in part, can be equivalently implemented in standard integratedcircuits, as one or more computer programs executed by one or morecomputers (e.g., as one or more programs running on one or more computersystems), as one or more programs executed by on one or more controllers(e.g., microcontrollers) as one or more programs executed by one or moreprocessors (e.g., microprocessors), as firmware, or as virtually anycombination thereof, and that designing the circuitry and/or writing thesource code for the software and or firmware would be well within theskill of one of ordinary skill in the art in light of the teachings ofthis disclosure. For example, those skilled in the relevant art canreadily create source based on the flowcharts of the figures herein,including FIGS. 5, and 7-16, and the detailed description providedherein.

As used herein processor-executable instructions and/or data can bestored on any non-transitory computer-readable storage medium, e.g.,memory or disk, for use by or in connection with any processor-relatedsystem or method. In the context of this specification, a“computer-readable storage medium” is one or more tangiblenon-transitory computer-readable storage medium or element that canstore processes-executable instruction and/or processor-readable dataassociated with and/or for use by systems, apparatus, device, and/ormethods described herein. The computer-readable storage medium can be,for example, but is not limited to, an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor system, apparatus, device,or articles of manufacture. Processor-executable instructions arereadable by a processor. More specific examples (a non-exhaustive list)of the computer-readable medium would include the following: a portablecomputer diskette (magnetic, compact flash card, secure digital, or thelike), a random access memory (RAM), a read-only memory (ROM), anerasable programmable read-only memory (EPROM, EEPROM, or Flash memory),a portable compact disc read-only memory (CDROM), digital tape, andother non-transitory storage media.

Many of the methods described herein can be performed with variations.For example, many of the methods may include additional acts, omit someacts, and/or perform acts in a different order than as illustrated ordescribed.

The various examples, implementations, and embodiments described abovecan be combined to provide further embodiments. Aspects of theembodiments can be modified, if necessary or desirable, to employsystems, circuits, devices, methods, and concepts in various patents,applications, and publications to provide yet further embodiments.

These and other changes can be made to the examples, implementations,and embodiments in light of the above-detailed description. In general,in the following claims, the terms used should not be construed to limitthe claims to the specific embodiments disclosed in the specificationand the claims, but should be construed to include all possibleembodiments along with the full scope of equivalents to which suchclaims are entitled. Accordingly, the claims are not limited by thedisclosure.

The various embodiments described above can be combined to providefurther embodiments. All of the commonly assigned US patent applicationpublications, US patent applications, foreign patents, foreign patentapplications and non-patent publications referred to in thisspecification and/or listed in the Application Data Sheet, including butnot limited to U.S. Provisional Application No. 62/255,899, filed Nov.16, 2015 are incorporated herein by reference, in their entirety.

1.-77. (canceled)
 78. A method of operation of an artificial neuralnetwork, including a plurality of variables, the plurality of variablesrepresenting a state of one or more units of the artificial neuralnetwork, a plurality of parameters, the plurality of parametersrepresenting weights of the one or more units and edges connecting theunits, and a plurality of buffers including a first buffer and a secondbuffer, the buffers each storing and updating a respective collection ofvalues for the plurality of variables and the plurality of parameters,the method comprising: computing, over a first period, by at least onehardware processor and using the first buffer, a first plurality ofvalues for the plurality of variables based, at least in part, on afirst plurality of values for the plurality of parameters; transferring,to the second buffer, by the at least one hardware processor, currentvalues for the plurality of variables at the end of the first periodthat have been computed by the at least one hardware processor and usingthe first buffer; calculating, over a second period, by the at least onehardware processor and using the first buffer, a first plurality ofincremental errors for the artificial neural network that occurred overthe first period; generating, by the at least one hardware processor, asecond plurality of values for the plurality of parameters based, atleast in part, on the first plurality of incremental errors; storing inthe first buffer, by the at least one hardware processor, the secondplurality of values for the plurality of parameters; interleaved withthe calculating of the first plurality of incremental errors using thefirst buffer over the second period, computing, over a third period, bythe at least one hardware processor and using the second buffer, asecond plurality of values for the plurality of variables based, atleast in part, on (a) the stored current values for the plurality ofvariables at the end of the first period and (b) the first plurality ofvalues for the plurality of parameters, wherein the second period atleast partially overlaps with the third period; transferring, to thefirst buffer, by the at least one hardware processor, current values forthe plurality of variables at the end of the third period that have beencomputed by the at least one hardware processor and using the secondbuffer; returning, by the at least one hardware processor, the secondplurality of values for the plurality of parameters; producing, by theat least one hardware processor, a plurality of output values for theartificial neural network; and causing, by the at least one hardwareprocessor, a robot to move based on the plurality of output values,wherein the robot is at least partially autonomous and partiallypiloted.
 79. The method of claim 78, further comprising: computing, bythe at least one hardware processor, over a fourth period, a thirdplurality of values for the plurality of variables in the artificialneural network based, at least in part, on the second plurality ofvalues for the plurality of parameters; calculating, by the at least onehardware processor, over a fifth period, a second plurality ofincremental errors for the artificial neural network that occurred overthe fourth period; generating, by the at least one hardware processor, athird plurality of values for the plurality of parameters; andinterleaved with calculating the second plurality of incremental errorsfor the artificial neural network, computing, by the at least onehardware processor, over a sixth period, a fourth plurality of valuesfor the plurality of variables for the artificial neural network based,at least in part, on the second plurality of values for the plurality ofparameters, wherein the fifth period at least partially overlaps withthe sixth period.
 80. The method of claim 78, further comprising:storing, by the at least one hardware processor, the first plurality ofvalues for the plurality of variables for the artificial neural network.81. The method of claim 78 wherein the artificial neural network isassociated with an objective function, the method further comprising:computing, by the at least one hardware processor, a gradient for theobjective function with respect to the plurality of parameters; andperforming, by the at least one hardware processor, gradient descentwith the gradient to create the second plurality of values for theplurality of parameters.
 82. The method of claim 81 wherein computingthe gradient for the objective function with respect to the parameterscomprises: unfolding, by the at least one hardware processor, theartificial neural network over a duration less than or equal to thefirst period; and performing, by the at least one hardware processor,backpropagation through time on the unfolded artificial neural network.83. The method of claim 78, further comprising: receiving, by the atleast one hardware processor, a plurality of input values for theartificial neural network.
 84. The method of claim 78, furthercomprising: computing, by the at least one hardware processor foranother artificial neural network that includes another plurality ofvariables and another plurality of parameters, a first plurality ofvalues for the other plurality of variables of the other artificialneural network based on a first plurality of values for the otherplurality of parameters; running, by the at least one hardwareprocessor, the other artificial neural network backward over the firstperiod; generating, by the at least one hardware processor, a secondplurality of values for the other plurality of parameters; andinterleaved with computing, for the other artificial neural the firstplurality of values for the other plurality of variables of the otherartificial neural network, computing, by the at least one hardwareprocessor, a second plurality of values for the other plurality ofvariables for the other artificial neural network over the third period.85. The method of claim 78 wherein behaviors of the artificial neuralnetwork are described by a system of differential equations, the methodfurther comprising: numerically integrating, by the at least onehardware processor, the system of differential equations.
 86. The methodof claim 78 wherein generating the second plurality of values for theplurality of parameters comprises: identifying, by the at least onehardware processor, a system model for the robot; and returning, by theat least one hardware processor, the system model for the robot.
 87. Asystem, comprising: at least one processor; a robot coupled to the atleast one processor; and at least one non-transitory computer-readablestorage media communicatively coupled to the at least one processor, andwhich stores at least one of processor-executable instructions or datathereon which when executed causes the at least one processor to:initialize implementation of an artificial neural network including aplurality of parameters and a plurality of variables; compute, over afirst period, a first plurality of values for the plurality of variablesbased, at least in part, on a first plurality of values for theplurality of parameters; store current values for the plurality ofvariables at the end of the first period in a second buffer; calculate,over a second period, by using a first buffer, a first plurality ofincremental errors for the artificial neural network that occurred overthe first period; generate a second plurality of values for theplurality of parameters based, at least in part, on the first pluralityof incremental errors for the artificial neural network; store thesecond plurality of values for the plurality of parameters in the firstbuffer; interleaved with the calculation of the first plurality ofincremental errors for the artificial neural network, compute, over athird period, by using the second buffer, a second plurality of valuesfor the plurality of variables based, at least in part, on (a) thestored current values for the plurality of variables at the end of thefirst period and (b) the first plurality of values for the plurality ofparameters, wherein the second period at least partially overlaps withthe third period; store current values for the plurality of variables atthe end of the third period that have been computed by using the secondbuffer to the first buffer; return the second plurality of values forthe plurality of parameters; produce a plurality of output values forthe artificial neural network; and cause the robot to move based on theplurality of output values, wherein the robot is an autonomous robot.88. The system of claim 87 wherein, when executed, theprocessor-executable instructions further cause the at least oneprocessor to: compute, over a fourth period, a third plurality of valuesfor the plurality of variables based, at least in part, on the secondplurality of values for the plurality of parameters; calculate, over afifth period, a second plurality of incremental errors for theartificial neural network that occurred over the fourth period; generatea third plurality of values for the plurality of parameters; andinterleaved with the calculation of the second plurality of incrementalerrors for the artificial neural network, compute, over a sixth period,a fourth plurality of values for the plurality of variables based, atleast in part, on the second plurality of values for the plurality ofparameters, wherein the fifth period at least partially overlaps withthe sixth period.
 89. The system of claim 87 wherein the artificialneural network is associated with an objective function, and whenexecuted, the processor-executable instructions further cause the atleast one processor to: compute a gradient for the objective functionwith respect to the plurality of parameters; and perform gradientdescent with the gradient creating the second plurality of values forthe plurality of parameters.
 90. The system of claim 89 wherein, whenexecuted, the processor-executable instructions further cause the atleast one processor to: unfold the artificial neural network over aduration less than or equal to the first period creating an unfoldedartificial neural network; and perform backpropagation through time onthe unfolded artificial neural network.
 91. The system of claim 87,further comprising: a sensor subsystem coupled to the at least oneprocessor; and where, when executed, the processor-executableinstructions further cause the at least one processor to: receive aplurality of input values for the artificial neural network from thesensor subsystem.
 92. The system of claim 87, further comprising: anetwork interface subsystem coupled to the at least one processor; andwhere, when executed, the processor-executable instructions furthercause the at least one processor to: send to at least one externalsystem, via the network interface subsystem, the plurality of variablesof the artificial neural network; and send to at least one externalsystem, via the network interface subsystem, the plurality of parametersof the artificial neural network.
 93. The system of claim 87, furthercomprising: a network interface subsystem coupled to the at least oneprocessor; and where, when executed, the processor-executableinstructions further cause the at least one processor to: receive theplurality of variables of the artificial neural network; and receive theplurality of parameters of the artificial neural network.
 94. One ormore non-transitory computer-readable media collectively storingcontents that, when executed by one or more processors, cause the one ormore processors to perform actions comprising: initializingimplementation of an artificial neural network including a plurality ofparameters and a plurality of variables; computing, over a first period,a first plurality of values for the plurality of variables based, atleast in part, on a first plurality of values for the plurality ofparameters; storing current values for the plurality of variables at theend of the first period in a second buffer; calculating, over a secondperiod, by using a first buffer, a first plurality of incremental errorsfor the artificial neural network that occurred over the first period;generating a second plurality of values for the plurality of parametersbased, at least in part, on the first plurality of incremental errorsfor the artificial neural network; storing the second plurality ofvalues for the plurality of parameters in the first buffer; interleavedwith the calculation of the first plurality of incremental errors forthe artificial neural network, computing, over a third period, by usingthe second buffer, a second plurality of values for the plurality ofvariables based, at least in part, on (a) the stored current values forthe plurality of variables at the end of the first period and (b) thefirst plurality of values for the plurality of parameters, wherein thesecond period at least partially overlaps with the third period; storingcurrent values for the plurality of variables at the end of the thirdperiod that have been computed by using the second buffer to the firstbuffer; returning the second plurality of values for the plurality ofparameters; producing a plurality of output values for the artificialneural network; and causing a robot to move based on the plurality ofoutput values, wherein the robot is at least partially autonomous. 95.The one or more non-transitory computer-readable media of claim 94wherein the actions further comprise: computing, over a fourth period, athird plurality of values for the plurality of variables based, at leastin part, on the second plurality of values for the plurality ofparameters; calculating, over a fifth period, a second plurality ofincremental errors for the artificial neural network that occurred overthe fourth period; generating a third plurality of values for theplurality of parameters; and interleaved with the calculation of thesecond plurality of incremental errors for the artificial neuralnetwork, computing, over a sixth period, a fourth plurality of valuesfor the plurality of variables based, at least in part, on the secondplurality of values for the plurality of parameters, wherein the fifthperiod at least partially overlaps with the sixth period.
 96. The one ormore non-transitory computer-readable media of claim 94 wherein theartificial neural network is associated with an objective function, andwherein the actions further comprise: computing a gradient for theobjective function with respect to the plurality of parameters; andperforming gradient descent with the gradient creating the secondplurality of values for the plurality of parameters.
 97. The one or morenon-transitory computer-readable media of claim 94 wherein the actionsfurther comprise: unfolding the artificial neural network over aduration less than or equal to the first period creating an unfoldedartificial neural network; and performing backpropagation through timeon the unfolded artificial neural network.